JSUM: A Multitask Learning Speech Recognition Model for Jointly Supervised and Unsupervised Learning

In recent years, the end-to-end speech recognition model has emerged as a popular alternative to traditional Deep Neural Network—Hidden Markov Model (DNN-HMM). This approach maps acoustic features directly onto text sequences via single network architecture, significantly streamlining construction process. However, training of models typically necessitates significant quantity supervised data achieve good performance, which poses challenge in low-resource conditions. The use unsupervised representation reduces this necessity. Recent research focused on techniques employing joint Connectionist Temporal Classification (CTC) and attention mechanisms, with some also concentrating presentation learning. paper proposes multi-task learning (JSUM). Our leverages pre-trained wav2vec 2.0 shared encoder that integrates CTC-Attention generative adversarial into unified architecture. method provides new language solution optimally utilizes datasets by combining CTC, attention, losses. Furthermore, our proposed is suitable for both monolingual cross-lingual scenarios.